Information processing device, data structure, information processing method, and non-transitory computer readable storage medium

ABSTRACT

An information processing device includes a communication unit that acquires first image data in which an observation value observed at a time t is used as a pixel value and a learning processing unit that generates second image data in which an observation value predicted to be observed at a time t+n after the time t is used as a pixel value from the first image data acquired by the acquiring unit based on a learning model obtained by machine learning using the first image data, in which the machine learning occurs based on a comparison of the first image data in which an observation value observed at a target time is used as a pixel value and the second image data in which an observation value predicted to be observed at the target time is used as a pixel value.

CROSS-REFERENCE TO RELATED APPLICATION

The present application claims priority to, and incorporates byreference, the entire contents of Japanese Patent Application No.2018-006575 filed in Japan on Jan. 18, 2018.

BACKGROUND 1. Field

Example implementations are directed to an information processingdevice, a data structure, an information processing method, and anon-transitory computer readable storage medium.

2. Related Art

A related art technique is directed to calculating a congestion degreein each of a plurality of past time zones of a target area as a timezone congestion degree with reference to past positioning information ofa mobile terminal in the target area, clustering the plurality of timezone congestion degrees, and generating a congestion degree pattern usedfor predicting a future congestion degree in the target area (forexample, see JP 2015-18336 A).

However, in the related technique, a future congestion degree isobtained for each spot or for each area, and thus there is a related artproblem in that a processing load increases, and data management iscomplicated. Further, such a related art problem is entirely a pointwhich is common to all application fields in which a certain observationvalue, such as the congestion degree, is associated with spatialcoordinates (e.g., arbitrary spatial coordinates), such as positioncoordinates on a map.

SUMMARY

According to one aspect of an example implementation, an informationprocessing device includes a communication unit configured to acquirefirst image data in which an observation value observed at a certaintime t is used as a pixel value. The information processing deviceincludes a learning processing unit that generates second image data inwhich an observation value predicted to be observed at a time t+n afterthe time t is used as a pixel value from the first image data acquiredby the communication unit on the basis of a learning model obtained bymachine learning using the first image data, wherein the machinelearning is machine learning based on a comparison of the first imagedata in which an observation value observed at a target time is used asa pixel value and the second image data in which an observation valuepredicted to be observed at the target time is used as a pixel value.

The above and other objects, features, advantages and technical andindustrial significance of this inventive concept will be betterunderstood by reading the following detailed description, whenconsidered in connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 a diagram illustrating an example of an information processingsystem 1 including an information processing device 200 in a firstexample implementation;

FIG. 2 is a diagram illustrating an example of a configuration of aninformation providing device 100 in the first example implementation;

FIG. 3 is a diagram illustrating an example of actual image data;

FIG. 4 is a diagram illustrating an example of a configuration of aninformation processing device 200 in the first example implementation;

FIG. 5 is a flowchart illustrating a flow of a series of processes whenan operation is performed by an information processing device sidecontrol unit 210;

FIG. 6 is a flowchart illustrating an example of a detailed flow of apredicted image data generation process;

FIG. 7 is a diagram illustrating an example of the PredNet 300;

FIG. 8 is a diagram illustrating an example of the PredNet 300;

FIG. 9 is a diagram in which actual image data and predicted image dataof respective periods are arranged and displayed;

FIG. 10 is a diagram illustrating an example of content provided fromthe information providing device 100 to a terminal device 10;

FIG. 11 is a diagram illustrating another example of content providedfrom the information providing device 100 to the terminal device 10;

FIG. 12 is a flowchart illustrating a flow of a series of processes whenlearning is performed by the information processing device 200;

FIG. 13 is a diagram schematically illustrating a flow of learning ofthe PredNet 300; and

FIG. 14 is a diagram illustrating an example of a hardware configurationof the information providing device 100 and the information processingdevice 200 of an example implementation.

DETAILED DESCRIPTION

An information processing device, a data structure, an informationprocessing method, and a program (e.g., non-transitory computer readablemedium including stored instructions executed by a microprocessor) willbe described with reference to the appended drawings.

Overview

An information processing device is realized by one or more processors.The information processing device obtains image data (hereinafterreferred to as “actual image data”) in which an observation valueobserved at a time t (e.g., a certain time) is used as a pixel value.The observation value is, for example, a value indicating a degree ofcongestion of people at a certain spot, i.e., a congestion degree. In acase in which the observation value is the congestion degree, the actualimage data may be indicated by a heat map in which color parameters suchas a hue, a saturation, and brightness (e.g., luminance) are used aspixel values in accordance with the size of congestion degree. Theactual image data is an example of “first image data” or “firstmulti-dimensional sequence data”.

If the actual image data is acquired, the information processing devicegenerates image data in which an observation value predicted to beobserved at a time t+n after a time t is used as a pixel value(hereinafter referred to as “predicted image data”), from the acquiredactual image data of the time t on the basis of a learning model thatperforms machine learning using the actual image data of another time t# (for example, a time of a period, such as a predetermined period,prior to the time t). The predicted image data is an example of “secondimage data” or “second multi-dimensional sequence data”.

For example, the learning model is obtained by machine learning based onan addition result of the actual image data at a certain time t # in thepast and the predicted image data at a time t # +n after the time t # inthe past. “n” indicates a period in which the observation value isobtained, for example, 1. The generated predicted image data has a dataformat similar to the actual image data. The term “similar” means that,for example, when the actual image data is a heat map in which a size ofa congestion degree is expressed by grayscale brightness, the predictedimage data is also a heat map in which a size of a congestion degree isexpressed by grayscale brightness. With this process, for example, it ispossible to predict a future observation value which is desired to befinally obtained on the basis of an image, without considering a featurequantity such as the presence or absence of a building in which peopleare likely to be crowded, a shape of a land, or a shape of a roadnetwork or a railroad network. As a result, it is possible to obtain afuture observation value suitably, for example, more easily at a lowerload.

FIRST EXAMPLE IMPLEMENTATION

Overall Structure

FIG. 1 is a diagram illustrating an example of an information processingsystem 1 including an information processing device 200 in the firstexample implementation. The information processing system 1 in the firstexample implementation includes, for example, one or more terminaldevices 10, an information providing device 100, and the informationprocessing device 200. These devices are connected to one another via anetwork NW.

Each device illustrated in FIG. 1 transmits and receives variousinformation via the network NW. Examples of the network NW include theInternet, a wide area network (WAN), a local area network (LAN), aprovider terminal, a wireless communication network, a wireless basestation, a dedicated line, and the like. All combinations of therespective devices illustrated in FIG. 1 need not be able to performcommunication with each other, and the network NW may optionally includea local network in part.

The terminal device 10 is a terminal device including an input device, adisplay device, a communication device, a storage device, and anarithmetic device such as a mobile phone such as a smartphone, a tabletterminal, or various kinds of personal computers. The communicationdevice includes a network card such as a network interface card (NIC), awireless communication module, and the like. The terminal device 10activates an agent such as a user agent (UA) such as a web browser or anapplication program (e.g., non-transitory computer readable mediumincluding stored instructions executed by a microprocessor) andtransmits a request corresponding to an input of a user to theinformation providing device 100. Further, the terminal device 10 whichhas started the UA displays various kinds of images on the displaydevice on the basis of information acquired from the informationproviding device 100.

The information providing device 100 is, for example, a web server thatprovides a web page to the terminal device 10 in response to a requestfrom a web browser. The web page includes content such as text, a stillimage, a moving image, or a sound. For example, in a case in which theweb page provides a map image, content includes a map image and a heatmap in which the congestion degree of people on the map is expressed bycolor. The information providing device 100 may be an application serverthat provides the above content to the terminal device 10 in response toa request from an application program (e.g., non-transitory computerreadable medium including stored instructions executed by amicroprocessor).

For example, the information processing device 200 acquires actual imagedata in which a congestion degree at a time t (e.g., a certain time) inan area (e.g., a certain area) is used as a pixel value from theinformation providing device 100, and generates predicted image data inwhich the congestion degree of a future time t+n in the area is used asa pixel value. Then, the information processing device 200 transmits thegenerated predicted image data to the information providing device 100which is an acquisition source of the actual image data. Upon receivingthe predicted image data, the information providing device 100 providesthe predicted image data of the future time t+n to the terminal device10 as content.

Configuration of Information Providing Device

FIG. 2 is a diagram illustrating an example of a configuration of theinformation providing device 100 in the first example implementation. Asillustrated, the information providing device 100 includes, for example,an information providing device side communication unit 102, aninformation providing device side control unit 110, and an informationproviding device side storage unit 130.

The information providing device side communication unit 102 includes,for example, a communication interface such as an NIC. The informationproviding device side communication unit 102 communicates with theterminal device 10 via the network NW, acquires a web browser requestand an application request from the terminal device 10, and acquiresposition information of the terminal device 10.

The information providing device side control unit 110 includes, forexample, a heat map generating unit 112 and a communication control unit114. These constituent elements are implemented, for example, such thata processor such as a central processing unit (CPU) executes a programstored in the information providing device side storage unit 130 (e.g.,non-transitory computer readable medium including stored instructionsexecuted by a microprocessor). Further, part or all of the components ofthe information providing device side control unit 110 may be realizedby hardware (e.g., circuitry) such as a large scale integration (LSI),an application specific integrated circuit (ASIC), a field-programmablegate array (FPGA), a graphics processing unit (GPU) or may be realizedby cooperation of software and hardware.

The information providing device side storage unit 130 is realized by astorage device such as a hard disc drive (HDD), a flash memory, anelectrically erasable programmable read only memory (EEPROM), a readonly memory (ROM), or a random access memory (RAM). In addition tovarious kinds of programs executed by a processor such as firmware andan application program (e.g., non-transitory computer readable mediumincluding stored instructions executed by a microprocessor), mapinformation 132, position information 134, and the like are stored inthe information providing device side storage unit 130. The mapinformation 132 includes, for example, a map image provided to theterminal device 10 as content and position coordinates of a building orthe like on the map. The position information 134 includes positioncoordinates of each terminal device 10 and an acquisition time of theposition coordinates.

The heat map generating unit 112 sequentially generates the actual imagedata indicated by the heat map at a period n, such as a predeterminedperiod (for example, 20 minutes). For example, the heat map generatingunit 112 extracts one or more position coordinates corresponding to atarget period from a plurality of position coordinates with reference tothe acquisition time of the position coordinates of the terminal device10. The heat map generating unit 112 maps the extracted positioncoordinates onto a map indicated by the map information 132 and derivesthe number of mapped position coordinates as the congestion degree ofpeople. Then, the heat map generating unit 112 generates data by causingthe heat map obtained by replacing the congestion degree with the pixelvalue to be superimposed on the map image indicated by the mapinformation 132 as the actual image data. The actual image data isindicated as, for example, a three-dimensional tensor (e.g., third-layertensor) data in which the congestion degree is associated with eachcoordinate of the map.

FIG. 3 is a diagram illustrating an example of the actual image data. Asin the example illustrated in FIG. 3, in the actual image data, the heatmap having a transmittance (e.g., predetermined transmittance) issuperimposed on the map image. The size of the pixel value of the heatmap, that is, the size of the congestion degree may be indicated by, forexample, a number of gradations (e.g., predetermined number ofgradations) among hues from pink to blue. As the actual image data isprovided to the terminal device 10 as the content, the user using theterminal device 10 can intuitively understand spots having congestion inaccordance with on a color depth, a hue, or the like.

The communication control unit 114 controls the information providingdevice side communication unit 102 such that the actual image datagenerated by the heat map generating unit 112 is transmitted to theterminal device 10 as content. Further, the communication control unit114 controls the information providing device side communication unit102 such that the actual image data generated by the heat map generatingunit 112 is transmitted to the information processing device 200.

Configuration of Information Processing Device

FIG. 4 is a diagram illustrating an example of a configuration of theinformation processing device 200 in the first example implementation.As illustrated in FIG. 4, the information processing device 200includes, for example, an information processing device sidecommunication unit 202, an information processing device side controlunit 210, and an information processing device side storage unit 230.

The information processing device side communication unit 202 includes,for example, a communication interface such as an NIC. The informationprocessing device side communication unit 202 communicates with theinformation providing device 100 via the network NW and acquires theactual image data from the information providing device 100. Theinformation processing device side communication unit 202 is an exampleof an “acquiring unit”.

The information processing device side control unit 210 includes, forexample, a learning processing unit 212 and a communication control unit214. These components are implemented, for example, such that aprocessor such as a CPU executes a program stored in the informationprocessing device side storage unit 230 (e.g., non-transitory computerreadable medium including stored instructions executed by amicroprocessor). Further, some or all of the components of theinformation processing device side control unit 210 may be realized byhardware (circuitry) such as an LSI, an ASIC, an FPGA, or a GPU or maybe realized by cooperation of software and hardware.

The information processing device side storage unit 230 is realized by astorage device such as an HDD, a flash memory, an EEPROM, a ROM, a RAM,or the like. The information processing device side storage unit 230stores deep learning model information 232 and the like in addition tovarious kinds of programs executed by a processor such as firmware andan application program (e.g., non-transitory computer readable mediumincluding stored instructions executed by a microprocessor).

The deep learning model information 232 is information specifying alearning model (e.g., a learning device) which is referred to by thelearning processing unit 212. The learning model is, for example, aPredNet 300. The PredNet 300 is a deep predictive coding network(s)which was derived from the principle of predictive coding inneuroscience and is realized by a plurality of neural networks,including but not limited to at least a deep convolutional recurrentneural network.

The deep learning model information 232 includes, for example, couplinginformation indicating how neurons (e.g., units) included in each of aninput layer, one or more hidden layers (e.g., intermediate layers), andan output layer constituting each neural network included in the PredNet300 (e.g., deep convolutional recurrent neural network) are coupled withone another and various kinds of information such as a couplingcoefficient assigned to data which is input or output between coupledneurons. The coupling information includes but is not limited to, forexample, information such as the number of neurons included in eachlayer, information designating a neuron with which each neuron iscoupled, an activation function for realizing each neuron, a gate formedbetween neurons in the hidden layer. The activation function forrealizing the neuron may be, for example, a normalized linear function(e.g., a ReLU function), a sigmoid function, a step function, any otherfunction, or the like. The gate selectively passes or weights datatransmitted between neurons in accordance with, for example, a value(for example, 1 or 0) returned by the activation function. The couplingcoefficient is a parameter of the activation function, and includes, forexample, a weight assigned to output data when data is output from aneuron of a certain layer to a neuron of a deeper layer in the hiddenlayer of the neural network. Further, the coupling coefficient mayinclude a bias component specific to each layer or the like.

The learning processing unit 212 generates (e.g., constructs) thePredNet 300 with reference to the deep learning model information 232,performs various kinds of calculations using the actual image dataacquired by the information processing device side communication unit202 as an input, and generates predicted image data.

The communication control unit 214 controls the information processingdevice side communication unit 202 such that the predicted image datagenerated by the learning processing unit 212 is transmitted to theinformation providing device 100.

Process Flow at Time of Operation

A flow of a series of processes when an operation is performed by theinformation processing device side control unit 210 will be describedbelow with reference to a flowchart. The term “when an operation isperformed” indicates a state in which a learning model learned to acertain extent by the learning processing unit 212 is used. FIG. 5 is aflowchart illustrating a flow of a series of processes when an operationis performed by the information processing device side control unit 210.A process of the present flowchart may be repeated.

First, the learning processing unit 212 is on standby until theinformation processing device side communication unit 202 acquires theactual image data from the information providing device 100 (S100), andwhen the actual image data is acquired by the information processingdevice side communication unit 202, the learning processing unit 212inputs the actual image data to the PredNet 300 (S102), and generatesthe predicted image data in which the congestion degree of a future timet+n which is one period n after the time t at which the actual imagedata is generated in the information providing device 100 is used as thepixel value (S104).

Then, the communication control unit 214 controls the informationprocessing device side communication unit 202 such that the predictedimage data of the time t+n generated by the learning processing unit 212is transmitted to the information providing device 100 (S106).Accordingly, the process of the present flowchart ends.

If the predicted image data is transmitted from the informationprocessing device 200, the information providing device 100 receives thepredicted image data and transmits the predicted image data to theterminal device 10 as content. As a result, the user using the terminaldevice 10 can understand a spot having congestion at the future timet+n.

FIG. 6 is a flowchart illustrating an example of a detailed flow of apredicted image data generation process. A process of the presentflowchart corresponds to the process of S104 described above.

First, the learning processing unit 212 generates the PredNet 300 withreference to the deep learning model information 232 (S200).

FIG. 7 and FIG. 8 are diagrams illustrating an example of the PredNet300. For example, the PredNet 300 includes two or more layers, eachlayer locally predicts an observation value or a feature quantityconstituting the observation value, and outputs a difference between aprediction result and an input to a subsequent layer. Each layerincludes an abstraction processing unit 310, a prediction generatingunit 320, a difference deriving unit 330, and a regression processingunit 340. Hereinafter, a processing result (output) of the abstractionprocessing unit 310 is referred to as “A_(l) ^(t)”, a processing result(output) of the prediction generating unit 320 is referred to as“A(hat)_(l) ^(t)”, a processing result (output) of the differencederiving unit 330 is referred to as “E_(l) ^(t)”, and a processingresult (output) of the regression processing unit 340 is referred to as“R_(l) ^(t)”. “(hat)” indicates a hat symbol of an alphabet letter A.

The abstraction processing unit 310 performs a process based on Formula(1) and outputs the processing result A_(l) ^(t). The abstractionprocessing unit 310 may be realized by, for example, a convolutionalneural network (CNN).

$\begin{matrix}{A_{l}^{t} = \left\{ \begin{matrix}x_{t} & {{{if}\mspace{14mu} l} = 0} \\{{MAXPOOL}\left( {{RELU}\left( {{CONV}\left( E_{l - 1}^{t} \right)} \right)} \right)} & {l > 0}\end{matrix} \right.} & (1)\end{matrix}$

For example, when a layer l of a processing target is 0, that is, whenthe abstraction processing unit 310 of a first layer is a target, theabstraction processing unit 310 outputs input actual image data x_(t) asthe processing result A_(l) ^(t) without change.

Further, when the layer l exceeds 0, that is, when the abstractionprocessing unit 310 of a subsequent layer is a target, the abstractionprocessing unit 310 performs a convolution process of a processingresult E_(l−1) ^(t) of the difference deriving unit 330 of animmediately previous layer and with a filter (e.g., predeterminedfilter). For example, when the layer l of the processing target is asecond layer, the abstraction processing unit 310 convolutes an imagewhich is the processing result E_(l−1) ^(t) of the difference derivingunit 330 and a filter (e.g., predetermined filter) (e.g., obtains aninner product), and compresses an image region including a plurality ofpixels overlapping with the filter into one unit region. A valueacquired by the convolution is associated with the unit region as afeature quantity. The abstraction processing unit 310 repeats theconvolution process while shifting a filter (e.g., predetermined filter)on the image and generates an image formed by a plurality of unitregions (hereinafter “convolutional layer”) (CONV(E_(l−1) ^(t))). Atthis time, the abstraction processing unit 310 performs padding at thetime of convolution, so that the convolutional layer has the same sizeas the image of the processing result E_(l−1) ^(t). The abstractionprocessing unit 310 couples the respective convolutional layers throughthe ReLU function (normalized linear function) (RELU(CONV(E_(l−1)^(t)))). The abstraction processing unit 310 compresses the size of eachof a plurality of coupled convolutional layers using a method called maxpooling (MAXPOOL(RELU(CONV(E_(l−1) ^(t))))), and abstracts the imagewhich is the processing result Then, the abstraction processing unit 310outputs the abstracted image as the processing result A_(l) ^(t).

The prediction generating unit 320 performs a process based on Formula(2) and outputs a processing result A(hat)_(l) ^(t). The predictiongenerating unit 320 may be realized by, for example, a convolutionalneural network.

Â _(l) ^(t)=RELU(CONV(R _(l) ^(t)))   (2)

For example, when the layer l of the processing target is the firstlayer, the prediction generating unit 320 convolutes the image which isthe processing result R_(l) ^(t) of the regression processing unit 340of the same layer and a filter (e.g., predetermined filter), andcompresses an image region including a plurality of pixels overlappingwith a filter (e.g., predetermined filter) into one unit region. Asdescribed above, the value acquired by convolution is associated withthe unit region as the feature quantity. The prediction generating unit320 repeats the convolution process while shifting a filter (e.g.,predetermined filter) on the image and generates a convolutional layerformed by a plurality of unit regions (CONV(R_(l) ^(t))). At this time,the prediction generating unit 320 performs padding at the time ofconvolution so that the convolutional layer has the same size as theimage of the processing result R_(l) ^(t). The prediction generatingunit 320 couples the respective convolutional layers through the ReLUfunction (RELU(CONV(E_(l−1) ^(t)))) and outputs the result as theprocessing result A(hat)_(l) ^(t). The processing result A(hat)_(l) ^(t)by the prediction generating unit 320 of the first layer indicates thepredicted image data.

The difference deriving unit 330 performs a process based on Formula (3)and outputs the processing result E_(l) ^(t).

E _(l) ^(t)=[RELU(A _(l) ^(t) −Â _(l) ^(t)); RELU(Â _(l) ^(t) −A _(l)^(t))]  (3)

For example, the difference deriving unit 330 calculates a difference(A_(l) ^(t)−A(hat)_(l) ^(t)) obtained by subtracting the processingresult A(hat)_(l) ^(t) from the processing result A_(l) ^(t) and adifference (A(hat)_(l) ^(t)−A_(l) ^(t)) obtained by subtracting theprocessing result A_(l) ^(t) from the processing result A(hat)_(l) ^(t),and outputs the output value of the ReLU function using the differencesas the processing result E_(l) ^(t).

The regression processing unit 340 performs a process based on Formula(4) and outputs the processing result R_(l) ^(t). For example, theprediction generating unit 320 may be realized by a combination of theconvolutional neural network and a recurrent network (e.g., a recurrentneural network (RNN)) in which a middle layer of the network is a longshort-term memory (LSTM) (hereinafter referred to as CONVLSTM). Forexample, the CONVLSTM is obtained by changing the inner product of aweight and a state variable in a convolution manner in a calculationformula of each gate of the LSTM.

R _(l) ^(t)=CONVLSTM(E _(l) ^(t−1) , R _(l) ^(t−1), UPSAMPLE(R _(l+1)^(t)))   (4)

For example, the regression processing unit 340 calculates the CONVLSTMon the basis of the previous processing result E_(l) ^(t−1) of thedifference deriving unit 330 temporarily stored in a memory region (anLSTM block) called a memory cell of the LSTM, its own previousprocessing result R_(l) ^(t−1), and the processing result R_(l+1) ^(t)of the regression processing unit 340 of the subsequent layer, andoutputs the calculation result as the processing result R_(l) ^(t). Whenthe size of the image which is the processing result R_(l+1) ^(t) of thesubsequent regression processing unit 340 is different from the size ofthe input actual image data x_(t), the regression processing unit 340performs up-sampling on the size of the input image as the processingresult R_(l+1) ^(t) from the subsequent stage so that it is adjusted tothe size of the actual image data x_(t). As a result, the sizes of theimages are unified in each target layer. Further, the regressionprocessing unit 340 causes the processing result R_(l) ^(t) which iscurrently acquired and the processing result E_(l) ^(t) which iscurrently acquired by the difference deriving unit 330 of the same layerto be stored in the memory cell of the LSTM.

If the PredNet 300 is generated, the learning processing unit 212determines whether or not the process of generating the predicted imagedata is an initial process (S202), and when it is determined that theprocess of generating the predicted image data is the initial process,the learning processing unit 212 sets the output value R_(l) ^(t) of theregression processing unit 340 of each layer and the output value E_(l)^(t) of the difference deriving unit 330 to the initial value (forexample, 0) and generates the predicted image data A(hat)_(l) ^(t)(S204).

For example, when the PredNet 300 includes a total of two layers, thatis, the l-th layer and a (l+1)-th layer after the l-th layer, and n is1, the regression processing unit 340 of the (l+1)-th layer sets anoutput value E_(l+1) ^(t−1) of a previous period t−1 by the previousdifference deriving unit 330 of the (l+1)-th layer to the initial value,sets the output value R_(l+1) ^(t−1) to the initial value since theoutput value R_(l+1) ^(t−1) of its own previous period t−1 is not storedin the memory cell of the LSTMs, and derives an output value R_(l+1)^(t) of the current period t. At this time, the regression processingunit 340 of the (l+1)-th layer causes the derived output value R_(l+1)^(t) of the current period t to be stored in the memory cell of theLSTM.

The regression processing unit 340 of the l-th layer sets the outputvalue E_(l) ^(t−1) of the previous period t−1 by the difference derivingunit 330 of the l-th layer to the initial value, sets the output valueR_(l) ^(t−1) to the initial value since the output value R_(l) ^(t−1) ofits own previous period t−1 is not stored in the memory cell of theLSTM, and derives the output value R_(l) ^(t) of the current period t onthe basis of the initial values E_(l) ^(t−1) and R_(l) ^(t−1) and theoutput value R_(l+1) ^(t) of the current period t derived by theregression processing unit 340 of the (l+1)-th layer. At this time, theregression processing unit 340 of the l-th layer causes the derivedoutput value R_(l) ^(t) of the current period t to be stored in thememory cell of the LSTM.

The prediction generating unit 320 of the l-th layer derives the outputvalue A(hat)_(l) ^(t) of the current period t on the basis of R_(l) ^(t)derived by the regression processing unit 340 of the l-th layer.Accordingly, the predicted image data A(hat)_(l) ^(t) of the currentperiod t is generated.

Then, the learning processing unit 212 calculates a difference betweenthe predicted image data A(hat)_(l) ^(t) generated on the basis of theinitial values R_(l) ^(t−1) and E_(l) ^(t−1) and the actual image datax_(t) acquired in the current period t (S206). For example, if theactual image data x_(t) of the period t is acquired by the informationprocessing device side communication unit 202, the learning processingunit 212 inputs the actual image data x_(t) of the period t to theabstraction processing unit 310 of the l-th layer. In the case of theinitial process, the abstraction processing unit 310 of the l-th layeroutputs the actual image data x_(t) acquired by the informationprocessing device side communication unit 202 to the difference derivingunit 330 of the l-th layer without change. In response to this, thedifference deriving unit 330 of the l-th layer derives the differenceE_(l) ^(t) corresponding to the current one period t on the basis of thepredicted image data A(hat)_(l) ^(t) generated by the predictiongenerating unit 320 of the l-th layer and the actual image data x_(t)output by the abstraction processing unit 310 of the l-th layer.

Then, the learning processing unit 212 derives a feature quantity of theimage from the difference the derived image data (S208). For example,the abstraction processing unit 310 of the (l+1)-th layer derives afeature quantity A_(l+1) ^(t) obtained by abstracting the image of theimage data of the current period t on the basis of the difference E_(l)^(t) derived by the difference deriving unit 330 of the l-th layer.

Then, the learning processing unit 212 derives a future feature quantityof an image (hereinafter referred to as a predicted feature quantity) onthe basis of the output value R_(i+1) ^(t) of the current period t(S210). For example, the prediction generating unit 320 of the (l+1)-thlayer derives a predicted feature quantity A(hat)_(l+1) ^(t) predictedto be obtained from the image data of the future period t+1 on the basisof the output value R_(l+1) ^(t) of the current period t derived by theregression processing unit 340 of the (l+1)-th layer. The predictedfeature quantity A(hat)_(l+1) ^(t) indicates the feature quantity of theimage data of the future period t+1 which is obtained only in thecurrent period t.

Then, the learning processing unit 212 calculates a difference betweenthe feature quantity A_(l+1) ^(t) acquired from the image data of thecurrent period t and the predicted feature quantity A(hat)_(l+1) ^(t)predicted to be obtained from the image data of the future period t+1(S212). For example, the difference deriving unit 330 of the (l+1)-thlayer obtains a difference (A_(l+1) ^(t)−A(hat)_(l+1) ^(t)) obtained bysubtracting the predicted feature quantity A(hat)_(l+1) ^(t) of thecurrent period t from the feature quantity A_(l+1) ^(t) of the currentperiod t and a difference (A(hat)_(l+1) ^(t)−A_(l+1) ^(t)) obtained bysubtracting the feature quantity A_(l+1) ^(t) of the current period tfrom the predicted feature quantity A(hat)_(l+1) ^(t) of the currentperiod t, and derives a difference E_(l+1) ^(t) of the feature quantityof the current period t using the differences as variables of the ReLUfunction. Accordingly, the process corresponding to one period in theinitial process ends.

On the other hand, when it is determined that the process of generatingthe predicted image data is not the initial process in the process ofS202, the learning processing unit 212 generates the predicted imagedata on the basis of the output values of the previous period of theregression processing unit 340 and the difference deriving unit 330 ofeach layer (S214).

For example, when the current period is t+1 which is a period advancedfrom the initial period t by one period, the regression processing unit340 of the (l+1)-th layer derives an output value R_(l+1) ^(t+1) thecurrent period t+1 on the basis of output value E_(l+1) ^(t) of theprevious period t by the difference deriving unit 330 of the (l+1)-thlayer (the processing result of S212) and the output value R_(l+1) ^(t)of the previous period t stored in the memory cell of the LSTM. At thistime, the regression processing unit 340 of the (l+1)-th layer causesthe derived output value R_(l+1) ^(t+1) of the current period t+1 to bestored in the memory cell of the LSTM.

The regression processing unit 340 of the l-th layer derives an outputvalue R_(l) ^(t+1) of the current period t+1 on the basis of the outputvalue E_(l) ^(t) of the previous period t by the difference derivingunit 330 of the l-th layer, the previous output value R_(l) ^(t) of theprevious period t stored in the memory cell of the LSTM, and the outputvalue R_(l+1) ^(t+1) of the current period t+1 derived by the regressionprocessing unit 340 of the (l+1)-th layer. At this time, the regressionprocessing unit 340 of the l-th layer causes the derived output valueR_(l) ^(t+1) of the current period t+1 to be stored in the memory cellof the LSTM.

The prediction generating unit 320 of the l-th layer derives an outputvalue A(hat)_(l) ^(t+1) of the current period t+1 on the basis of R_(l)^(t+1) derived by the regression processing unit 340 of the l-th layer.Accordingly, the predicted image data A(hat)_(l) ^(t+1) of the currentperiod t+1 is generated.

Then, the learning processing unit 212 abstracts actual image datax_(t+1) acquired in the current period t+1 by the convolution processand the max pooling as a process of S206, and derives a differencebetween the predicted image data A(hat)_(l) ^(t+1) generated on thebasis of R_(l) ^(t) and E_(l) ^(t) which are derivation results of theprevious period t and the abstracted actual image data x_(t+1).

Then, the learning processing unit 212 derives a feature quantityA_(l+1) ^(t+1) acquired by abstracting the image data of the currentperiod t+1 on the basis of the derived difference E_(l) ^(t+1) of theimage data as a process of S208.

Then, the learning processing unit 212 derives a predicted featurequantity A(hat)_(l+1) ^(t+1) predicted to be obtained from image data ofa future period t+2 on the basis of the regression processing resultR_(l+1) ^(t+1) of the current period t+1 as a process of S210.

Then, the learning processing unit 212 derives a difference E_(l+1)^(t+1) between the feature quantity A_(l+1) ^(t+1) acquired from theimage data of the current period t+1 and the predicted feature quantityA(hat)_(l+1) ^(t+1) predicted to be obtained from the image data of thefuture period t+2 as a process of S212. Accordingly, the processcorresponding to one period in the second and subsequent processes ends.

FIG. 9 is a diagram in which the actual image data and the predictedimage data of respective periods are arranged and displayed. In FIG. 9,t1 represents an initial period, and it means that predicted image datahas never been generated. For example, if actual image data x_(t1) isacquired at a time point of the period t1, the learning processing unit212 inputs the actual image data x_(t1) to the PredNet 300, andgenerates predicted image data A(hat)_(l) ^(t1) predicted to be obtainedat a time point of a period t2. Further, if actual image data x_(t2) isacquired at a time point of the period t2, the learning processing unit212 inputs the actual image data x_(t2) to the PredNet 300, generatespredicted image data A(hat)^(t2) predicted to be obtained at a timepoint of a period t3. As described above, the period is repeated, andpredicted image data after one period is generated each time actualimage data is acquired. Accordingly, the predicted image data issequentially transmitted from the information processing device 200 tothe information providing device 100.

FIG. 10 is a diagram illustrating an example of content provided fromthe information providing device 100 to the terminal device 10. In acase in which the information providing device 100 continuouslytransmits the actual image data of the respective periods t1 to t9sequentially to the information processing device 200 to generate thepredicted image data, the information processing device 200 generatespredicted image data A(hat)^(t9) of a period t10 next to a period t9,and transmits the predicted image data A(hat)^(t9) to the informationproviding device 100. Upon receiving the predicted image dataA(hat)^(t9) from the information processing device 200, the informationproviding device 100 provides content including the actual image datax_(t1) to x_(t9) which has been continuously transmitted to theinformation processing device 200 so far and the predicted image dataA(hat)^(t9) obtained from predicting actual image data x_(t10) of aperiod t10 to the terminal device 10. Accordingly, the congestion degreeto the present and the future congestion degree are displayed on ascreen of terminal device 10 as the heat map.

In the above example, the information processing device 200 has beendescribed as generating the predicted image data in which the congestiondegree of a future time (period) t+n which is one period n after thetime (period) t at which the actual image data is generated is used asthe pixel value, but not limited thereto. For example, the informationprocessing device 200 may predicts the predicted image data of thefuture time (period) t+n after one period n and the predicted image dataof the future time t+kn which is one or more periods n after the futuretime (period) t+n. “n” indicates a period as described above, and “k”indicates a natural number (e.g., an arbitrary natural number)indicating the number of future periods to be predicted.

For example, in a case in which the current period is t+1, even beforeor when the actual image data x_(t+1) is acquired by the informationprocessing device side communication unit 202, the abstractionprocessing unit 310 of the l-th layer which is a first layer regards thepredicted image data A(hat)^(t) generated in the previous period t bythe prediction generating unit 320 of the l-th layer as the actual imagedata x_(t+1), and abstracts the actual image data x_(t+1) by performingthe convolution process and the max pooling. Upon receiving it, thedifference deriving unit 330 of the l-th layer derives the differenceE_(l) ^(t+1) of the current period t+1 on the basis of the predictedimage data A(hat)^(t+1) generated by the prediction generating unit 320of the l-th layer and the predicted image data A(hat)^(t) abstracted bythe abstraction processing unit 310 of the l-th layer. The constituentelement of the (l+1)-th layer subsequent to the l-th layer obtains thefeature quantity A_(l+1) ^(t+1) of the current period t+1 on the basisof the difference E_(l) ^(t+1) between the two pieces of predicted imagedata, and derives the difference E_(l+1) ^(t+1) between the featurequantity A_(l+1) ^(t+1) and the predicted feature quantity A(hat)_(l+1)^(t+1). The prediction generating unit 320 of the l-th layer generatespredicted image data A(hat)^(t+2) predicting the actual image datax_(t+2) of the period t+2 on the basis of the difference E_(l+1) ^(t+1).As described above, instead of the actual image data x_(t+1) obtained inthe current period t+1, the actual image data x_(t+2) of the futureperiod t+2 is further predicted using the predicted image dataA(hat)^(t) predicted as the actual image data x_(t+1) of the currentperiod t+1 at a past time point, and thus it is possible to provide theuser with the heat map indicating the future congestion degree as thecontent. Further, future actual image data x_(t+k) after k or moreperiods (for example, k≥2) may be predicted, and in this case, it ispossible to provide the user with the heat map indicating the futurecongestion degree as the content.

FIG. 11 is a diagram illustrating another example of content providedfrom the information providing device 100 to the terminal device 10. Forexample, in a case in which the current period is t5, when the futureactual image data after 5 periods is predicted, the informationprocessing device 200 transmits predicted image data A(hat)_(l) ^(t6) toA(hat)^(t10) of periods t6 to t10 to the information providing device100. Upon receiving it, as illustrated in FIG. 11, the informationproviding device 100 provides content including actual image data x_(t1)to x_(t5) up to the current period t5 and the predicted image dataA(hat)^(t6) to A(hat)^(t10) corresponding to the future 5 periods to theterminal device 10. Accordingly, it is possible to further improve theconvenience of the user as compared with a case in which a futurecongestion degree after one period is displayed as the heat map.

Process Flow at Time of Learning

A flow of a series of processes when learning is performed by theinformation processing device 200 will be described below with referenceto flowcharts. The term “when learning is performed” indicates a statein which a learning model used at the time of operation, that is, thePredNet 300 is learned. FIG. 12 is a flowchart illustrating a flow of aseries of processes when learning is performed by the informationprocessing device 200. A process of the present flowchart may berepeated, for example, at a period (e.g., predetermined period).

First, if the actual image data x_(t) is acquired by the informationprocessing device side communication unit 202 in the current period t,the learning processing unit 212 inputs actual image data x_(t) to thePredNet 300 (S300), and generates the predicted image data A(hat)^(t) inwhich the congestion degree of the future time t+n which is one period nafter the time t at which the actual image data x_(t) is generated inthe information providing device 100 is used as the pixel value (S302).

Then, the learning processing unit 212 derives a sum y(=x_(t)+A(hat)^(t)) of the actual image data x_(t) acquired at the timepoint of the period t and the predicted image data A(hat)^(t) generatedat the time point of the period t (S304).

Then, if the current period is t+n, and the actual image data x_(t+n) isacquired by the information processing device side communication unit202, the learning processing unit 212 derives a difference ΔE betweenthe actual image data x_(t+n) acquired at the time point of the periodt+n and the sum y of the actual image data x_(t) and the predicted imagedata A(hat)^(t) at the time point of the period t (S306).

Then, the learning processing unit 212 learns the PredNet 300 so thatthe derived difference ΔE is minimized using an error back propagationtechnique or a stochastic gradient descent technique (S308). Forexample, the learning processing unit 212 determines (e.g., decides) theparameter of the CONVLSTM which realizes the regression processing unit340 of each layer so that the difference ΔE is minimized. Examples ofthe parameter of the CONVLSTM include an input weight, a recurrentweight, a peephole weight, and a bias weight. Further, the learningprocessing unit 212 may determine (e.g., decide) some or all of theparameters of the abstraction processing unit 310, the predictiongenerating unit 320, and the difference deriving unit 330. The learningprocessing unit 212 updates the parameter of the PredNet 300 in the deeplearning model information 232 with the parameter determines (e.g.,decided) as described above. Accordingly, the process of the presentflowchart ends.

FIG. 13 is a diagram schematically illustrating a flow of learning ofthe PredNet 300. As illustrated in FIG. 13, if the actual image datax_(t) is input to the PredNet 300, the predicted image data A(hat)^(t)predicting the actual image data x_(t+n) of the period t+n after theperiod t is generated. The learning processing unit 212 obtains the sumy of the predicted image data A(hat)^(t) output from the predictiongenerating unit 320 of the l-th layer of the PredNet 300 and the actualimage data x_(t) used when the predicted image data A(hat)^(t) isgenerated, and uses the sum y as the output of the PredNet 300. Then,when the next processing period t+n arrives, the learning processingunit 212 learns the PredNet 300 on the basis of the difference ΔEbetween the output y of the period t and the actual image data x_(t+n)acquired from the information providing device 100 in the period t+n,that is, the input x_(t+n) of the period t+n. As described above, theactual image data obtained in the same period (that is, the actual imagedata input to the PredNet 300 in the same period to generate thepredicted image data) is added to the predicted image data generated ineach period while the difference between the output value of thesubsequent layer of the PredNet 300 and the output value of thepreceding layer is gradually reduced, and the PredNet 300 is learned asthe supervised learning model so that the difference ΔE between the sumy of the actual image data of the same period which is the additionresult and the predicted image data and the actual image data input tothe PredNet 300 in the next period is minimized, and thus it is possibleto generate the predicted image data with higher accuracy at the time ofoperation.

According to the first example implementation described above, theinformation processing device side communication unit 202 that acquiresthe actual image data x_(t) in which the observation value observed at acertain time t is used as the pixel value and the learning processingunit 212 that generates the predicted image data A(hat)_(l) ^(t) inwhich the observation value predicted to be observed at a time t+n afterthe time t is used as the pixel value from the actual image data x_(t)of the time t on the basis of the PredNet 300 learned in advance by deeplearning using the actual image data x are provided, and the learningprocessing unit 212 learns the PredNet 300 on the basis of a comparisonof actual image data x, obtained at a certain target time T andpredicted image data A(hat)_(l) ^(t) in which an observation valuepredicted to be obtained at the target time T is used as the pixelvalue, and it is possible to obtain a future observation valueappropriately.

SECOND EXAMPLE IMPLEMENTATION

A second example implementation will be described below. In the firstexample implementation described above, the actual image data x input tothe PredNet 300 has been described as being the three-dimensional tensordata in which the congestion degree is associated with each coordinateof the map. On the other hand, the second example implementation differsfrom the first example implementation in that the actual image data xinput to the PredNet 300 is four-or more dimensional tensor data. Thefollowing description will proceed focusing on the difference from thefirst example implementation, and description of points common to thefirst example implementation will be omitted. In the description of thesecond example implementation, the same parts as those in the firstexample implementation are denoted by the same reference numerals.

For example, the actual image data x in the second exampleimplementation may be four-dimensional tensor data in which thecongestion degree is associated with each coordinate of the map and isassociated with the acquisition time of the position information usedwhen the congestion degree is derived. In other words, the actual imagedata x may be multi-dimensional sequence data in which a sequence ofeach coordinate axis, a sequence of congestion degrees, and a sequenceof acquisition times are combined. The acquisition time of the positioninformation is an example of the “measurement time of the observationvalue”.

The learning processing unit 212 of the second example implementationlearns the PredNet 300 in advance on the basis of the actual image datax when the actual image data x is the four-dimensional tensor data.Accordingly, it is possible to generate the predicted image data inwhich the future congestion degree is used as the pixel value for eachtime or each time zone at the time of operation. Further, since theactual image data x is the four-dimensional tensor data including theacquisition time of the position information, it is possible to generatethe predicted image data in which the future congestion degree is usedas the pixel value for each day of the week, each day, or each season.Accordingly, for example, it is possible to generate the predicted imagedata of a specific day of the week (for example, Friday), a specificseason, or a specific holiday (for example, New Year's Eve, New Year'sDay, or the like) can be generated.

Further, the observation value included in the actual image data x asthe pixel value is not limited to the congestion degree of people andmay be the congestion degree of other moving bodies such as automobilesor airplanes or may be a certain value which changes with the passage oftime such as temperature, or humidity, rainfall, concentration of fineparticulate matters such as pollens, or the like.

According to the second example implementation described above, sincethe PredNet 300 is learned on the basis of the actual image data whichis the four-or more multi-dimensional sequence data, it is possible togenerate the predicted image data which is the four-or moremulti-dimensional sequence data at the time of operation. Accordingly,if time information is included in the actual image data, the timeinformation is included in the predicted image data as well, and thuswhen the content including the predicted image data is provided to theterminal device 10, the user using terminal device 10 can understand atime zone in which congestion is likely to occur. Accordingly, theconvenience of the user using the terminal device 10 can be furtherimproved.

Hardware Configuration

The information providing device 100 and the information processingdevice 200 of the example implementations described above are realizedby, for example, a hardware configuration illustrated in FIG. 14. FIG.14 is a diagram illustrating an example of a hardware configuration ofthe information providing device 100 and the information processingdevice 200 according to an example implementation.

The information providing device 100 has a configuration in which an NIC100-1, a CPU 100-2, a RAM 100-3, a ROM 100-4, a secondary storage device100-5 such as a flash memory or an HDD, and a drive device 100-6 areconnected to one another via an internal bus or a dedicatedcommunication line. A portable storage medium such as an optical disk isloaded onto the drive device 100-6. A program (e.g., non-transitorycomputer readable medium including stored instructions executed by amicroprocessor) stored in the secondary storage device 100-5 or aportable storage medium loaded onto the drive device 100-6 is extractedonto the RAM 100-3 by a DMA controller (not illustrated) or the like andexecuted by the CPU 100-2, so that the information providing device sidecontrol unit 110 is realized. The program referred to by the informationproviding device side control unit 110 may be downloaded from anotherdevice via the network NW.

The information processing device 200 has a configuration in which anNIC 200-1, a CPU 200-2, a RAM 200-3, a ROM 200-4, a secondary storagedevice 200-5 such as a flash memory or an HDD, and a drive device 200-6are connected to one another via an internal bus or a dedicatedcommunication line. A portable storage medium such as an optical disk isloaded onto the drive device 200-6. A program (e.g., non-transitorycomputer readable medium including stored instructions executed by amicroprocessor) stored in the secondary storage device 200-5 or aportable storage medium loaded onto the drive device 200-6 is extractedonto the RAM 200-3 by a DMA controller (not illustrated) or the like andexecuted by the CPU 200-2, so that the information processing deviceside control unit 210 is realized. The program (e.g., non-transitorycomputer readable medium including stored instructions executed by amicroprocessor) referred to by the information processing device sidecontrol unit 210 may be downloaded from another device via the networkNW.

According to one aspect, a future observation value can be obtained moresuitably.

Although the inventive concept has been described with respect tospecific example implementations for a complete and clear disclosure,the appended claims are not to be thus limited but are to be construedas embodying all modifications and alternative constructions that mayoccur to one skilled in the art that fairly fall within the basicteaching herein set forth.

What is claimed is:
 1. An information processing device, comprising: acommunication unit that acquires first image data in which anobservation value observed at a time t is used as a pixel value; and acontrol unit that generates second image data in which an observationvalue predicted to be observed at a time t+n after the time t is used asa pixel value from the first image data acquired by the communicationunit based on a learning model obtained by machine learning using thefirst image data, wherein the machine learning occurs based on acomparison of the first image data in which an observation valueobserved at a target time is used as a pixel value, and the second imagedata in which an observation value predicted to be observed at thetarget time is used as a pixel value.
 2. The information processingdevice according to claim 1, wherein the machine learning comprises deeplearning.
 3. The information processing device according to claim 2,wherein the learning model comprises a PredNet.
 4. The informationprocessing device according to claim 1, wherein the communication unitfurther acquires first image data in which an observation value observedat the time t+n is used as a pixel value, and when the first image datain which the observation value observed at the time t+n is used as thepixel value is acquired by the communication unit, the control unitgenerates new second image data in which an observation value predictedto be observed at a time t+kn after the time t+n is used as a pixelvalue from the first image data based on the learning model.
 5. Theinformation processing device according to claim 1, wherein the controlunit generates new second image data in which an observation valuepredicted to be observed at a time t+kn after the time t+n is used as apixel value from the second image data in which the observation valuepredicted to be observed at the time t+n is used as a pixel value basedon the learning model.
 6. The information processing device according toclaim 1, wherein a measurement time of the observation value is furtheradded to the first image data as an element.
 7. The informationprocessing device according to claim 1, wherein the observation value isa congestion degree indicating a degree of congestion of one or moreusers.
 8. The information processing device according to claim 1,wherein the learning processing unit: derives a sum of the second imagedata in which the observation value predicted to be observed at thetarget time is used as a pixel value and the first image data of a timeprior to the target time, and determines a parameter of the learningmodel by the machine learning based on a difference between the derivedsum and the first image data in which the observation value observed atthe target time is used as a pixel value.
 9. An information processingdevice, comprising: a communication unit that acquires firstmulti-dimensional sequence data including an observation value observedat a time t as one sequence; and a control unit that generates secondmulti-dimensional sequence data including an observation value predictedto be observed at a time t+n after the time t as another sequence fromthe first multi-dimensional sequence data acquired by the acquiring unitbased on a learning model acquired by machine learning using the firstmulti-dimensional sequence data, wherein the machine learning occursbased on a comparison of the first multi-dimensional sequence dataincluding an observation value observed at a certain target time as theone sequence and the second multi-dimensional sequence data including anobservation value predicted to be observed at the target time as theanother sequence.
 10. A data storage and retrieval system for a computermemory, comprising: a data structure comprising a parameter that isdetermined based on a sum of first multi-dimensional sequence dataincluding an observation value observed at a certain time t as onesequence and second multi-dimensional sequence data including anobservation value predicted to be observed at a time t+n after the timet as another sequence.
 11. The data structure according to claim 10,further comprising: the parameter determined based on a differencebetween the sum and first multi-dimensional sequence data including anobservation value observed at the time t+n as the one sequence.
 12. Acomputer-implemented information processing method, the methodcomprising: acquiring, by a computer, first image data in which anobservation value observed at a time t is used as a pixel value; andgenerating, by the computer, second image data in which an observationvalue predicted to be observed at a time t+n after the time t is used asa pixel value from the acquired first image data based on a learningmodel obtained by machine learning using the first image data, whereinthe machine learning occurs based on a comparison of the first imagedata in which an observation value observed at a target time is used asa pixel value and the second image data in which an observation valuepredicted to be observed at the target time is used as a pixel value.13. A non-transitory computer-readable storage medium having storedtherein one or more executable instructions, the one or moreinstructions comprising: acquiring first image data in which anobservation value observed at a time t is used as a pixel value; andgenerating second image data in which an observation value predicted tobe observed at a time t+n after the time t is used as a pixel value fromthe acquired first image data based on a learning model obtained bymachine learning using the first image data, wherein the machinelearning occurs based on a comparison of the first image data in whichan observation value observed at a target time is used as a pixel valueand the second image data in which an observation value predicted to beobserved at the target time is used as a pixel value.
 14. Thenon-transitory computer-readable storage medium of claim 13, wherein themachine learning comprises deep learning.
 15. The non-transitorycomputer-readable storage medium of claim 14, wherein the learning modelcomprises a PredNet.
 16. The non-transitory computer-readable storagemedium of claim 13, wherein the acquiring further comprises thecommunication unit acquiring first image data in which an observationvalue observed at the time t+n is used as a pixel value, and when thefirst image data in which the observation value observed at the time t+nis used as the pixel value is acquired by the communication unit, thegenerating comprises the control unit generating new second image datain which an observation value predicted to be observed at a time t+knafter the time t+n is used as a pixel value from the first image databased on the learning model.
 17. The non-transitory computer-readablestorage medium of claim 13, wherein the generating comprises the controlunit generating new second image data in which an observation valuepredicted to be observed at a time t+kn after the time t+n is used as apixel value from the second image data in which the observation valuepredicted to be observed at the time t+n is used as a pixel value basedon the learning model.
 18. The non-transitory computer-readable storagemedium of claim 13, further comprising adding a measurement time of theobservation value to the first image data as an element.
 19. Thenon-transitory computer-readable storage medium of claim 13, wherein theobservation value is a congestion degree indicating a degree ofcongestion of one or more users.
 20. The non-transitorycomputer-readable storage medium of claim 13, the generating furthercomprising: deriving a sum of the second image data in which theobservation value predicted to be observed at the target time is used asa pixel value and the first image data of a time prior to the targettime, and determining a parameter of the learning model by the machinelearning based on a difference between the derived sum and the firstimage data in which the observation value observed at the target time isused as a pixel value.